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The LCROSS (Lunar Crater Observation and Sensing Satellite) project presented a 
number of challenges to the preparation for mission operations. A class D mission under 
NASA’s risk tolerance scale, LCROSS was governed by a $79 million cost cap and a 29 
month schedule from “authority to proceed” to flight readiness. LCROSS was NASA Ames 
Research Center’s flagship mission in its return to spacecraft flight operations after many 
years of pursuing other strategic goals. As such, ARC needed to restore and update its 
mission support infrastructure, and in parallel, the LCROSS project had to newly define 
operational practices and to select and train a flight team combining experienced operators 
and staff from other arenas of ARC research. This paper describes the LCROSS flight team 
development process, which deeply involved team members in spacecraft and ground system 
design, implementation and test; leveraged collaborations with strategic partners; and 
conducted extensive testing and rehearsals that scaled in realism and complexity in 
coordination with ground system and spacecraft development. As a testament to the 
approach, LCROSS successfully met its full mission objectives, despite many in-flight 
challenges, with its impact on the lunar south pole on October 9, 2009. 
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NGTS = Northrop Grumman Technical Services, Lanham, MD 

ORT = operational readiness test 

ROC = Remote Operations Center 

SOC = Science Operations Center (NASA ARC) 

TT = thread test 


I. Introduction 

The preparation for LCROSS mission operations was particularly challenging due to the low LCROSS cost cap, 
a fast-paced development schedule, the unique and non-repeating nature of the mission, and the number of years that 
had passed since NASA ARC had flown a similar mission. After years pursuing strategic priorities outside the flight 
operations realm, NASA ARC had to undertake a significant restoration of ground data systems (GDS) and 
infrastructure. Meanwhile, LCROSS could not draw its mission operations team from a standing cadre of recently- 
exercised ARC mission operations staff. 

Despite these challenges, NASA ARC also entered with a number of advantages, namely its prior flight 
operations heritage and experience, collected via earlier missions and operations support conducted at NASA ARC 
and outside the Center; its strong staff covering a diverse set of disciplines, including spacecraft engineering, wind 
tunnel operations, computer science and software engineering; and, without a pre-existing culture of large-scale 
mission operations, the institutional freedom to tailor an operations philosophy to the lean LCROSS project model. 

The remaining subsections within in Section I 
summarize the conditions under which LCROSS 
team development began. Section II describes how 
team development advanced in parallel with GDS and 
spacecraft development, and Sections III through VI 
detail the various means our team used to develop 
and tailor operational practices to LCROSS, and to 
train for flight. Then, Section VII provides a brief 
overview of LCROSS flight experience and team 
performance, from the human perspective, as a 
complement to descriptions of our experience with 
the spacecraft contained in Ref TBD and TBD. We 
evaluate our development approach, and consider its 
value for future missions in Section VIII. 

A. LCROSS Project 

LCROSS was conceived as an economical 
approach to determining the nature of hydrogen 
observed in permanently shadowed craters at the 
lunar poles. LCROSS was selected in April 2006, 
under the Lunar Precursor Robotic Program (LPRP) of NASA’s Exploration Systems Mission Directorate (ESMD), 
as a secondary payload to be launched with the Lunar Reconnaissance Orbiter (LRO). Under the earliest possible 
launch date of October 28, 2008, LCROSS was obligated to meet a 29 month development schedule from ATP 
through launch readiness 1 '. The LCROSS project was constrained to a $79 million cost cap, of which $TBD million 
was designated for mission operations. The project’s tight budget and fast-paced schedule provided a strong 
impetus for utilizing a small staff in creative ways to accomplish the huge body of work required to prepare for 
flight. 

To ease the budget and schedule challenges, NASA designated LCROSS as “Class D” under its risk tolerance 
scale. This classification allowed the project to accept greater risk relative to higher-profile, large-budget missions, 
and shaped the development strategy and operational approach. However, the definition of Class D was not well- 
documented, nor did it have a strong precedence in prior NASA missions. Hence, the LCROSS project team had to 
work out how the classification translated into actual programmatic, engineering and operational practice, in 
coordination with (and often with the approval of) NASA ARC, the program office and NASA headquarters. 



Figure 1. Flight Team in the Mission Operations 
Control Room (MOCR). The MOCR seated seven 
operators and was the focal point for execution and 
monitoring operations during the mission. 


11 Though the LRO/LCROSS launch date ultimately slipped to June 18, 2009, LCROSS met the original deadline 
for flight readiness in October 2008. 
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Importantly, it had to balance the need to reduce cost and save time against the natural tendency to pursue the 
reliability standards of bigger-budget missions. 

B. Initial State of Mission Operations 

The LCROSS mission operations campaign was the first NASA ARC had led since the Lunar Prospector (LP) 
mission, which ended in 1999 1 . In the intervening years between LP and LCROSS, while ARC pursued a number of 
other high-priority strategic goals for NASA, the ARC mission operations infrastructure was specialized for other 
mission domains, including Space Shuttle payload operations support, and in other cases was not actively 
maintained. NASA ARC was concurrently preparing for the Kepler mission 2 . However, spacecraft operations for 
that mission were to be conducted outside NASA ARC, and science operations were being performed in a separate 
NASA ARC facility, and followed a very different operational model. 

Another consequence of the many years between flight missions at ARC was the minimal number of available, 
recently-exercised operations staff from which an LCROSS team could be selected. Though some team members 
had flight operations experience, many came from other backgrounds and were transformed into experienced team 
members through LCROSS. 

Without a continuous history of space operations at ARC, and without an active set of practices that are typically 
carried from one team to the next in an active mission operations facility, the LCROSS team also needed to define or 
borrow the full range of generic operational practices. In retrospect, this provided both a disadvantage and an 
advantage. In one respect, this added a large body of work to the team beyond LCROSS-specific preparations. 
However, the absence of a pre-defined set of practices allowed the team to scale its process to the needs and 
constraints of LCROSS. 


II. Team Development Process 

With little operational culture to drawn from early in the LCROSS project, the Mission Operations System 
(MOS) team had to define everything about how it would operate. Broadly, the MOS needed to define its 
composition and its body of operational practices, both general (e.g. voice loop protocols, Deep Space Network 
interaction, telemetry data archival, anomaly resolution processes) and LCROSS-specific (including the mission 
plan, team roles and responsibilities, team interactions and data transfers, command product generation and 
verification, procedures, flight rules, etc). Through a team training and test plan, which it also needed to invent, it 
would prepare its operators for flight. 

On the selection of LCROSS, NASA ARC initiated a mission operations facilities and ground systems 
restoration effort that continued in parallel with LCROSS spacecraft and mission operations team development. 
Hence, the team, spacecraft, and ground system grew together. The MOS was responsible for defining requirements 
for mission operations facilities and the GDS, whose implementation was handled under project-external ARC 
funding. Details of the ground systems development are described in Hunt et al. (Ref TBD). 

LCROSS adopted an iterative approach to both MOS and GDS development, recognizing that without key 
elements in place and lacking realism in early tests and simulations, operational practices and even team 
composition might require modification. Training occurred on a gradually-developing ground system target, 
sometimes with serious limitations in capability, until late in the development schedule when all elements were in 
place. MOS process definitions were refined gradually through collaboration and repeated simulations. 

Of course LCROSS relied heavily on the experiences of earlier mission operations teams. It selected from 
previously-successful operational practices wherever possible, as long as they were relevant to the LCROSS mission 
and fit within the lean LCROSS model. Given the team’s experience mix, it took specific inspiration from Lunar 
Prospector, the Mars Exploration Rover (MER) mission. Space Shuttle and International Space Station (ISS) 
operations, commercial communications satellite operations, prior Northrop Grumman missions, and even from 
ARC wind tunnel operational practices. 


III. Composing the Team 

The LCROSS mission plan was characterized by two focused periods of intense operations separated by months 
of lower-intensity spacecraft monitoring and occasional critical events. The MOS team structure reflected the need 
to efficiently support both high- and low-intensity operations. The MOS was built on a light weight core team at 
NASA ARC, supplemented with key maneuver design and navigation expertise from JPL and GSFC that could 
support most operational roles for the mission. For the most intense periods and other critical events, the MOS also 
included an extended team of spacecraft engineers from NG to provide real-time, deep expertise in the behavior of 
spacecraft systems. 
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A. The Core Team at ARC 

As a starting point, the LCROSS MOS team adopted a team structure and roles typical of other orbiting science 
missions (see Figure TBD). However, LCROSS budget limitations dictated a small and efficient MOS staff. 
LCROSS selected a core team of individuals to fill the roles of each MOS subsystem, but in most cases the staff 
count was barely enough to fill the needs of the role (1-2 people each subsystem), and did not allow for a backup 
operator. In some cases, given a particular mix of skill and experience, staff members were assigned to more than 
one role. Furthermore, LCROSS could not afford the luxury of employing fully-independent operations and GDS 
development teams. In fact, many operators on the team began work on LCROSS as GDS developers and later 
inherited additional responsibilities as mission operators. Importantly, early in LCROSS development, we 
recognized the benefit of including members of the science and payload engineering teams in operations. The role 
of Payload Engineer was added to the MOS, and was represented by a console position (one of seven) in the Mission 
Operations Control Room (MOCR). 

Prior to joining LCROSS, NASA ARC flight team members came from a diverse set of backgrounds. Some 
served in other mission operations roles, including in support of Lunar Prospector, Gravity Probe B, and other 
orbital science missions; the Mars Exploration Rover mission; Space Shuttle and International Space Station 
operations; and commercial satellite missions. Others were employed as engineers in support of space and defense- 
related engineering projects. A number of staff once led or supported wind tunnel operations at NASA ARC, and 
many had been researchers or software developers in the fields of autonomous systems, artificial intelligence and 
robotics. Regardless of background, all team members started as technically very competent in their respective 
fields, and all were very excited and motivated by the prospect of supporting operations for a lunar impact mission. 

As we describe later, concurrent with operations training, LCROSS operators served in a wide range of design, 
development and test roles both for the ground system and the spacecraft itself. Despite the extra work and split 
assignments, that experience was advantageous, and highly relevant to operations. 

B. External Team Members 

Beyond the small core team of MOS developer/operators at NASA ARC, the LCROSS team partnered with 
other organizations to bolster ARC experience in key areas, to create a reserve capacity for key mission phases, and 
to support spacecraft anomalies, should they occur. 

Due to the particular challenges of LCROSS trajectory planning and navigation for precise lunar impact 
targeting, NASA ARC augmented its own expertise in these areas with partners from the Jet Propulsion Laboratory 
(JPL) (providing navigation and orbit propagation expertise) and NASA Goddard Space Flight Center (GSFC) 
(providing complementary trajectory and maneuver planning capabilities). JPL also assigned a small team to 
LCROSS devoted to Deep Space Network (DSN) communications link scheduling. 

The MOS also needed to possess deep spacecraft systems and subsystems expertise to assess spacecraft 
performance throughout the mission, and to diagnose and suggest remedies for anomalous spacecraft behavior, as 
necessary. As a small project, LCROSS employed only two ARC spacecraft engineers - the Project Systems 
Engineer and his deputy - to oversee spacecraft development at Northrop Grumman. Both were ultimately assigned 
as Systems Engineers on the flight team. However, with only two ARC LCROSS spacecraft engineers to call on, 
and lacking the time to independently train a new set of ARC staff in the detailed design of LCROSS hardware and 
software, a natural approach was to augment the MOS with the team of Northrop Grumman spacecraft systems and 
subsystem engineers who were serving as LCROSS lead spacecraft designers and integration and test engineers (see 
Figure TBD). There were no better candidates in terms of depth of knowledge, and many were excited at the 
prospect of participating in operations. Two Northrop Grumman units were involved in LCROSS development: 
Aerospace Systems in Redondo Beach, CA (NGAS), and Technical Services in Lanham, MD (NGTS). Engineers 
were drawn from both into the MOS. 

Despite the clear benefits, this approach presented a number of additional challenges: 

1) LCROSS would have to devote budgetary resources to train these engineers (through classes, tests and 
rehearsals, etc), many of whom were neither experienced in operations generally, nor in the LCROSS 
concept of operations specifically. 

2) Because these engineers also served in leading roles on LCROSS spacecraft development, they were 
unlikely to be able to support all operational training activities (concurrent with spacecraft testing). 

3) The spacecraft engineering team needed facilities and equipment to allow them to interact in real time with 
telemetry. To co-locate them at ARC would entail a substantial increase in operational floor space, an 
enormous travel budget, and a significant travel burden on each of the participants (with high attrition 
likely). A distributed solution would require the build-up of dedicated remote operations rooms at each of 
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the Northrop Grumman facilities, and would introduce the difficulties of distributed team coordination 
during rehearsals and flight. 

4) The LCROSS project could not afford to employ the full set of spacecraft engineers full-time for the entire 
flight phase. Part-time involvement required that engineers work on other projects. During flight, without 
management support and careful staff planning, these engineers were at risk of being fully claimed by other 
Northrop Grumman projects. 

C. Team Collaboration during Development and Test Phases 

The core MOS team comprised the members at ARC, the Navigation Lead at JPL, and two Northrop Grumman 
systems engineers. The ARC team shared some management responsibility with lead members at JPL and Northrop 
Grumman for coordinating their teammates. 

A key to MOS success was its tight integration and culture of open communications across organizations. Small 
team size enhanced the MOS’s ability to communicate and to remain focused and coordinated. Most MOS 
development and coordination meetings were open to the entire “core” team, and often to the broader team. 

All organizations demonstrated full commitment to the LCROSS project. Of critical importance, management at 
all participating organizations successfully established an atmosphere of collaboration, without barriers. 
Furthermore, inter-organizational relationships were not strictly bounded by standard contract limitations. Northrop 
Grumman openly voiced problems encountered during spacecraft development, and involved ARC at every stage in 
working through these obstacles. As full-fledged MOS team members, Northrop Grumman engineers regularly 
reviewed and steered the development of spacecraft operational procedures at ARC and, through participation in 
procedure development and training exercises, they could also better understand the operational effects of remaining 
spacecraft design decisions. As described in Section VI, ARC MOS engineers were also included in key aspects of 
spacecraft development and test, further solidifying this collaborative bond. Finally, tight cross-organizational team 
integration allowed external participants to communicate to each other effectively. For example, the JPL Navigation 
team worked closely with Northrop Grumman attitude control engineers to develop models of the perturbation 
effects of various controller modes on the trajectory. Through this body of interactions, and because of the tone set 
by management early in the project, the MOS (and the project at large) became a cohesive unit concerned 
principally with mission success, despite political and organizational boundaries. 

In balance, baselining a distributed MOS presented other challenges. Though the team was well-integrated 
across organizational lines, communications breakdowns and isolation sometimes occurred due to physical 
separation. With the bulk of the team at ARC, informal discussions there occasionally unintentionally excluded 
external partners. Even full-team teleconferences were prone to poor audio quality and acoustics or occasionally 
poor network or graphics-sharing software performance. The importance of effective distributed team coordination 
was recognized early in the MOS development, and was reflected both in MOS operational processes and in the 
GDS design (voice loops, video sharing, improved teleconferencing equipment, etc.). 

IV. Development of Operational Practices 

LCROSS operational practices were built on standard models of space operations, tailored to the LCROSS 
mission plan (see Ref TBD for a description of flight operations). The most fundamental of these practices are 
expressed in the choice of team organizational roles, described in the previous section. How these team members 
operated together is the subject of this section. The full set of practices is too large to describe in detail here, but the 
following sections highlight some of the salient features of LCROSS practice. 

A. Workflow and Shift Scheduling 

LCROSS operated under a typical workflow model with four phases: Planning, Command Generation and 
Verification, Execution, and Assessment. Because the LCROSS mission was largely absent of routine operations, 
the workflow strongly emphasized the first two phases. Because LCROSS operated at close range to Earth, 
Execution and much of Assessment could be conducted in near real-time. A full cycle, depending on the operation, 
lasted anywhere from 7 hours to two days: 

1) Planning: During this phase, the Mission and Maneuver Design team designed trajectory correction 
maneuvers and spacecraft attitude changes to meet the requirements of impact targeting, science instrument 
calibrations, communications, and other activities, while minimizing propellant consumption. Payload 
Engineers and Science Team members designed specific payload observation sequences, as applicable, and 
Systems Engineers designated housekeeping activities to be executed. Link Schedulers coordinated Deep 
Space Network contact periods for LCROSS, and Link Analysts predicted link performance for future 
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contacts based on past experience. Planning phase culminated in an Activity Selection Review, during 
which the team reviewed maneuver designs and selected and ordered the supporting activities to be 
undertaken during a future contact. The resulting fully-ordered list of activities and their associated 
parameters was called an Activity Plan. The Mission and Maneuver Design Engineers operated in multiple 
overlapping shifts during peak activity periods; all others were individuals operating in a single shift. 

2) Command Generation and Verification: During this phase, the Command Sequencing Engineer converted 
the Activity Plan into a Command Plan, combining onboard command sequences and ground-based 
commanding procedures. This process was partially automated to minimize human error, but was 
sufficiently flexible to enable in-flight modifications. The team verified the correctness and safety of the 
command products using a combination of analysis (e.g. rudimentary automated flight rule checking) and 
simulation. The LCROSS simulator combined ground copies of spacecraft avionics, ran spacecraft flight 
software, and simulated spacecraft dynamic behavior. This phase concluded with the Command Approval 
Meeting, during which the team reviewed all Command Plan elements and associated analysis and 
simulation results before providing final approval to proceed. The minimal team supporting Command 
Generation and Verification comprised a single shift of activity. 

3) Execution: In this phase, the MOS enacted the Command Plan. All execution was conducted by the team 
in near real-time (with speed-of-light delays of seconds) and typically resulted in the collection of all 
associated telemetry for concurrent or subsequent review. Execution was closely orchestrated by a Flight 
Director, and all commands were sent exclusively by a Flight Controller. During periods of 24-hour contact 
with the spacecraft, the Execution team was divided into two shifts, A and B, each staffed with operators 
for every core role. Often one shift actively monitored the spacecraft during Planning and Command 
Generation phases, and the other performed Execution. During periods with sporadic spacecraft contacts, 
single Execution shifts conducted the operations for each given DSN pass. Science team operators 
augmented the standard Execution team for events focused on science data collection. 

4) Assessment: This phase ran in parallel with Execution phase and persisted while the team estimated the 
trajectory, characterized orbit perturbations and evaluated burn performance (Navigation), analyzed 
engineering telemetry (spacecraft and payload engineers), and analyzed science data (science team) to infer 
spacecraft health status and to determine the degree of success for maneuvers and science activities. This 
information was fed back into the next Planning phase, and to occasional status briefings for project 
stakeholders. 

Day-to-day operations were led by the Flight Director on duty. The Mission Operations Manager ensured that 
the flight team adhered to accepted operational practices, helped coordinate during anomaly responses, and was the 
primary communicator between the flight team and project management and LCROSS stakeholders (NASA ARC, 
LPRP program, and NASA headquarters). 

B. Facilities, Physical Distribution and Team Communications 

The organization of LCROSS facilities was strongly influenced by the workflow design and the mix of internal 
and external MOS team members. Operations were centered at ARC, and utilized three primary rooms, all of which 
were built up and equipped in stages during LCROSS development (for technical details on facility and GDS design, 
please consult Ref TBD). Mission Operations Control Room (MOCR) activities focused on the execution of 
spacecraft commands and spacecraft monitoring (Execution and Assessment Phases). The MOCR was a self- 
contained facility that accommodated a core set of seven operators with telemetry and command workstations and 
primary and backup telemetry and commanding data systems. The Science Operations Center (SOC) was the room 
from which the Science Team evaluated science instrument data streaming in real-time from the spacecraft during 
science events, coordinated the ground telescope observation campaign at lunar impact, and performed some of the 
post-event science data analysis. The SOC was separated physically from the MOCR to permit science-oriented 
discussions and parallel activities without disturbance to the MOCR team. The Mission Support Room (MSR) was 
the focus of activity during the Planning and Command Generation and Verification Phases, and also portions of 
Assessment Phase. It was designed as a center for off-line trajectory planning, development and test of command 
products, and discussion of engineering issues (including anomaly resolution). The MSR was separated from the 
MOCR and SOC to allow discussions and coordination in parallel with real-time and science activities. 

Outside of ARC, Northrop Grumman assembled and maintained two Remote Operations Centers (ROCs), one at 
the NGAS facility in California, and another at NGTS in Maryland. These supported two roles. First, the NG 
ROC’s acted as an extension of the MOCR, providing real-time oversight of subsystem and system behavior during 
Execution phases. Second, they acted as engineering extensions of the MSR for more detailed review and 
discussion of spacecraft behavior (both nominal and anomalous), with easy access to NG design and test repositories 
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and factory support (Assessment). Most of the subsystem engineering disciplines and the Systems Engineers 
operated from NGAS, while the subsystems engineers overseeing Avionics and Flight Software operated from 
NGTS. The JPL Navigation and Link Scheduling teams operated from their own fully-equipped facilities, while 
GSFC Mission Design team partners supported the critical initial and final weeks of the mission from the MSR at 
ARC, but worked from their home facilities during the months of Cruise Phase. 

A major challenge was enabling 
effective communications and situational 
awareness over this distributed team. We 
discovered in early tests that standard 
voice communications, on their own, 
were insufficient to coordinate the team 
and to keep operators informed on 
minute-by-minute progress during 
Execution or in reviewing planning and 
command generation results. Achieving 
good communications and situational 
awareness became a major effort in 
LCROSS GDS development (TBD ref 
here). Many of these elements were 
introduced gradually into operations over 
the LCROSS buildup. 

The primary means of 
communications for Execution phase 
remained a voice loop system that linked 
all operations consoles as well as 
operators at the launch site and the Deep 
Space Network. Multiple channels 
allowed the team to compartmentalize 
communications according to purpose 
and criticality. For Activity Selection 
Reviews and Command Approval 
Meetings (see Section TBD), the team 
collaborated via conference telephones 
and a secure meeting system that allowed 



Figure 2. LCROSS Flight Team and Physical Distribution. Small 
boxes represent the full set of flight team staff while encompassing 
boxes represent facilities from which they most often worked. Planning 
and Command Generation & Verification Phases were concentrated in 
the MSR. Execution Phase was led by the MOCR, often with assistance 
from Systems Engineers at NG. Critical events were also supported by 
subsystems engineers, up to the full set of disciplines, depending on 
need. The SOC and Payload Engineers participated in all 
science/payload activities. 

network-based distribution of graphical presentations. All team workstations were networked to a Mission Data 
Product Server that served as the central repository and transfer point for data products. 

For situational awareness, operators at all facilities were equipped with workstations to provide access to real- 
time telemetry. Workstations mirrored the display of the single commanding workstation in the MOCR to enable 
viewing of commands as they were being issued to the spacecraft. A video distribution system and large overhead 
monitors enabled rooms at ARC to mirror workstation displays in other ARC rooms. For real-time operations, ARC 
distributed a display of the current procedure step to all facilities, and also a real-time animation of the LCROSS 
spacecraft and trajectory that was keyed off of telemetry to depict the attitude state of the vehicle and fields-of-view 
of all sensors. ARC also provided software to depict the detailed LCROSS event timeline, which was synchronized 
with UTC to show the current activities, time since past events and time until future events. 


C. Tight integration of Science/Payload with MOS 

LCROSS science operations were characterized by focused activities lasting up to one hour '' (but separated by 
hours, days or weeks), during which the science team planned to collect image and spectroscopy data, assess 
instrument performance and make parameter adjustments in near real-time. Over the course of development, the 
MOS came to recognize that without tight coordination with the science and payload teams both prior to and during 
flight, it could not necessarily ensure that it would meet the needs of the science team during these events, or 
establish and maintain a science operations concept that was consistent with the rest of operations. 

Over the development period, the MOS endeavored to make scientists and Payload Engineers integral to 
operations. Scientists and Payload Engineers became required signatories for the approval of any command product 


" Thermal constraints dictated that the LCROSS payload could not be powered on for more than one hour at a time. 
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involving the instrument payload. For Execution Phase, the Payload Engineer was assigned one of seven consoles 
in the MOCR and led the assessment of payload engineering performance during all payload activities. The SOC 
was equipped with science workstations networked to the MOS-wide telemetry data server, enabling scientists to 
view and assess instrument data seconds after capture on the spacecraft. To keep real-time commanding authority 
consistent with other operations, science team members and the Payload Engineer made command requests (from a 
pre-arranged set) to the Flight Director verbally over the voice loop. The Flight Director quickly evaluated the 
criticality of science requests relative to other immediate ground-commanded tasks, and passed them on to the Flight 
Controller for radiation. This preserved a notion of centralized authority at the “big picture” level, albeit with some 
added delay. In cases where time was especially critical, the MOS devised more immediate responses to specific 
science team requests. 

The science and payload teams worked closely with the MOS over many months to develop command sequences 
and procedures to address the full range of anticipated science activities. To test these designs and ultimately to 
train for flight, the same team members were integral participants in all science-oriented MOS tests and rehearsals. 


V. Operational Training and Test Program 

The LCROSS MOS training program included a combination of conventional and less conventional training 
methods. This section describes all of the more conventional approaches we used to ready ourselves for flight. In 
addition to attending class-based training, operators were encouraged to attend spacecraft design reviews through 
Critical Design Review, and were obligated to attend operations-oriented seminars and workshops focusing on 
specific spacecraft subsystems and mission events. Operator certification was not formalized, but was judged 
informally by peers and MOS leadership over the course of many system-wide tests and rehearsals. 

A. Ground Data System Training 

Most training in the use of GDS tools (telemetry and commanding software, voice loops, link analysis, etc) was 
performed conventionally, either through formal classes taught by software vendors, through classes taught by key 
MOS team members to the broader team, or informal instruction between team members followed by on-the-job 
practice. Many GDS tools were custom designed and implemented for LCROSS, and in some cases the developers 
were the end users of the software. Partial-team and system-wide tests provided the team with many opportunities 
to practice GDS tool usage. 

B. S/C Reviews as Training Opportunities 

An important part of training was to familiarize team members with the spacecraft design. The MOS recognized 
spacecraft design reviews as valuable training opportunities, and required its membership to attend detailed 
subsystem Critical Design Audits presented by Northrop Grumman. The ARC Project Systems Engineer and his 
deputy (also MOS Systems Engineers) interacted closely with NG throughout the spacecraft development and test, 
and consequently became two of the most knowledgeable MOS team members with regards to spacecraft design. 

C. Workshops/Seminars 

To complement training via spacecraft design reviews, the MOS held a series of Spacecraft Subsystem 
Workshops, led by an NG lead subsystem engineer (and often assigned to the MOS). The team conducted 
workshops for the Attitude Control, Avionics, Flight Software, Power, Propulsion, Communications, Autonomy and 
Fault Management, and Payload subsystems. Each workshop lasted from four to eight hours and reviewed 
subsystem design, but emphasized subsystem operation and upkeep, including the use of primary relevant 
commands and telemetry, typical on-orbit behavior, troubleshooting and operational constraints. All ARC MOS 
members were required to attend these workshops. 

A series Operational Focus Workshops, led by the Lead Flight Director, provided in-depth reviews of operations 
for specific mission events at the system level. These reviewed the sequences of events, DSN utilization, pre- 
planned command sequences, operational procedures, the workflow covering the events, timelines and staffing. 
Workshops were held for Activation and Checkout; Cruise Phase Housekeeping; Cruise Phase Trajectory Correction 
Maneuvers; Star Field Calibration and Lunar Swingby; Earth Look Calibrations; and Separation and Lunar Impact. 
All MOS team members were required to attend these workshops. 

In conjunction with the Operational Focus Workshops, the MOS conducted many workshops to develop and 
review command sequences for the same events. These meetings gathered spacecraft subsystem and systems-level 
expertise, Flight Directors and Flight Controllers, and for science-related activities, members of the science and 
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payload team to communicate design requirements to the Activity Planning & Sequencing Lead, who implemented 
the sequences and, for parameterized activities, tools to be used to generate the sequences during flight. 

D. MOS Operational Tests 

With the concurrent development of the MOS, operational facilities and GDS, MOS training was limited by the 
degree of availability of facilities and tools, as governed by the GDS development schedule. The LCROSS team 
recognized two things: first, that MOS training could not wait until the facilities and GDS were fully deployed, and 
second, that the GDS team could not confidently deploy a quality product without intermediate validation testing by 
the MOS team. Hence, the MOS coordinated with the GDS development team to create a schedule of interleaved 
GDS releases and MOS operational tests (of procedures and processes) that also served as training exercises and 
GDS requirement validation tests. The GDS release schedule was designed to provide increasingly capable sets of 
hardware, software and command products that built logically upon previous releases, but that were usable in 
intermediate form to support MOS tests. MOS tests grew in sophistication and realism, starting with tests of 
contiguous “threads” of the operational workflow, and growing to system-wide tests closely approximating the 
operations for full mission events. MOS tests were designed with full knowledge of, and limited by, the capabilities 
of each GDS release. Similarly, the scope of GDS releases was driven in part by the needs of upcoming MOS tests. 

An important piece of the 
GDS delivery was the 
LCROSS spacecraft 

simulator. It combined a 
partial copy of LCROSS 
avionics hardware, a full copy 
of flight software, and a 
software-based dynamic 
simulator to simulate vehicle 
flight dynamics and the 
behavior of other spacecraft 
systems not represented in 
hardware (e.g. IRU, star 
tracker, power electronics and 
solar array, thruster modules). 
Importantly, the simulator 
was built from pieces of the 
simulator used by NG to 
develop LCROSS flight 
software. Therefore, MOS 
validation tests simulating the 
behavior of the spacecraft 
could not be conducted until 
the completion of main-line 
flight software development. (TBD: is this correct, or did NG give up pieces of their sim much earlier than FSW 
delivery?) However, once transitioned to the MOS, the LCROSS spacecraft simulator became a dedicated resource 
for GDS development (for command products) and for MOS tests (during which it represented the spacecraft). The 
simulator was critical to the success of preparations for LCROSS. 

Early tests were designed and conducted by the MOS Flight Team Lead, who also was the Lead Flight Director. 
As tests became more complex, the MOS recognized that it needed a dedicated Test Conductor to design test 
scenarios and to steer the course of events of tests (including operation of the LCROSS spacecraft simulator) behind 
the scenes. The Test Conductor was selected for his extensive prior experience with mission operations and 
spacecraft systems engineering. Hence, after being trained in the specifics of the LCROSS mission and spacecraft 
and operation of the simulator, the Test Conductor could comfortably lead system- wide tests and independently 
judge the performance of the MOS team. 

MOS tests of operational “threads” were called Thread Tests (TTs), and began with tests of telemetry receipt 
(telemetry decoding, distribution and display), commanding, and data product routing and archival, to name a few. 
Later TTs tested more significant threads, for example planning and command generation for trajectory correction 
maneuvers. In addition to supporting GDS validation tests, TTs helped train operators in the use of GDS tools, in 



Figure 3. Interleaved GDS and MOS Development Cycles to Support 
Parallel Development. GDS cycles provided increasing system-wide 
capabilities. MOS cycles focused on specific mission events and drove 
development of event-specific GDS elements. Operationally-realistic MOS tests 
doubled as GDS requirements validation tests; results occasionally prompted the 
refinement of GDS requirements. In general, GDS releases served multiple MOS 
development cycles. 
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the use of realistic basis data to create and deliver mission data products, and also to accustom them to segments of 
the MOS workflow. 

With the availability of all ARC operations rooms and sufficient GDS readiness, the MOS introduced system- 
level Engineering Readiness Tests (ERTs) that brought a significant portion of the team together and tested the 
complete workflow in support of specific mission events. The primary goal of ERTs was to prove that all ground 
elements were ready to support specific mission events, while team readiness was secondary. Hence, ERTs allowed 
intermediate stops and starts, timeline stretching and compression, and event re-ordering to meet team schedule 
constraints and to focus on the most important aspects of the test. ERTs sometimes exercised contingency execution 
paths, but always warned the MOS team in advance. 

Once the GDS and MOS processes were fully tested at least once for a particular event, the MOS conducted an 
Operational Readiness Test (ORT) for the same event. ORTs were like ERTs, except that they adhered more strictly 
to the mission timeline (duration and ordering), and focused on validating MOS processes and team readiness over 
GDS tools. ORTs also tested contingency paths, but without warning the MOS team. Being able to correctly plan, 
execute and assess a mission event within the constraints of the mission timeline was a key validation of tools and 
processes, and a strong indication of operational readiness. 

Rehearsals were conducted as final training events before launch and also during flight. Unlike ORTs, 
rehearsals typically exercised two or more days of continuous operation (active and inactive periods), mimicking a 
segment of the mission timeline to the minute. The Test Conductor ran rehearsals with the highest possible fidelity, 
and the team was expected to exercise every MOS process exactly how it would in flight. Rehearsals also exercised 
ancillary support activities like planning catered meals for the team during 24-hour operations, and securing and 
using on-site lodging for any operator living beyond a maximum range to avoid the dangers of driving after long 
shifts. The MOS conducted a First Week Rehearsal prior to launch covering launch through Lunar Swingby (six 
days in duration), and a Last Two Days rehearsal during flight covering the final trajectory correction maneuver, 
Centaur separation and lunar impact (two days in duration). A significant result from these tests is that team 
members grew to fully appreciate the rigors of the mission timeline, and discovered their own strengths and 
weaknesses in adapting to unusual sleep and waking hours while supporting critical events. Furthermore, with all 
team members involved in around-the-clock operations, with shifts on opposing sleep schedules, team members 
came to understand the importance of good inter-shift communications, when the full team was available. (TBD: 
other rehearsals? I’ve already forgotten!) 

Special tests: Flatsat and End-to-End Testing 

VI. Training Through Development and Test 

In addition to the more conventional training approaches, many of the MOS team enhanced their training 
through participation in spacecraft, payload and GDS development and testing. Many of these assignments were set 
up explicitly as training engagements, while in other cases, staff started as developers for the project, and evolved 
into operators. In the active design, test and review of the spacecraft and GDS, operators became far better 
acquainted with mission systems than would have been possible by classes and rehearsals alone. Dual 
responsibilities also had their share of disadvantages. 

A. MOS Team in Spacecraft Development 

As discussed earlier, the ARC Project Systems Engineer (PSE) and Deputy PSE were both assigned as lead 
Systems Engineers on the MOS. Furthermore, the MOS enlisted many NG LCROSS spacecraft engineers to 
augment the core ARC team. Many of these NG staff members held lead subsystems or systems design roles, or 
were lead integration and test engineers. Essentially, the MOS benefitted from the inherent training each of these 
engineers acquired over more than two years of deep involvement with LCROSS design, construction and test. 
Maintaining the engineering team through the entire flight saved the project significantly in training time and 
coordination effort. The MOS trained these engineers in operational practices (in the same training pipeline as for 
the rest of the MOS) far more easily than it could have trained general spacecraft engineers in the intricacies of 
LCROSS systems design and test behaviors (given the much greater body of knowledge, but without a training 
program in place). 

B. MOS Team in LCROSS/LRO Testing 

Under the theory that MOS team members could not possibly build a sufficiently deep understanding of 
LCROSS spacecraft designs and operation by attending reviews and studying documentation, the LCROSS project 
negotiated with Northrop Grumman to embed two team members into the spacecraft test flow. One served on-site at 
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C. MOS Team Members as Developers 
of Onboard Command Sequences 

In another collaboration between the 
MOS and Northrop Grumman spacecraft 
development, the team member designated 
as Command Sequencing Engineer (the 
operator in charge of implementing MOS 
command sequences before and during 


NASA GSFC as a technical liaison between the LRO and LCROSS projects, focusing on common hardware 
development and test. He also served at NGTS as an avionics and flight software test engineer during avionics unit 
testing and integrated “flatsaf ’ testing. The second served on-site at NGAS as a liaison and interface between NASA 
ARC and NG systems engineers for science payload integration, the S/C build and S/C integration and test (I&T). 
He also served as an interface between the I&T team, split between NG and the launch facility, and the MOS at 
NASA ARC during the conductance of S/C end-to-end testing. 

As a result of their assignments, these 
operators became two of the most 
knowledgeable on the team in the detailed 
operation of the spacecraft. Both were 
assigned as primary Flight Controllers, the 
operators directly responsible for sending 
commands to, and acquiring telemetry 
from, the spacecraft. However, during 
their remote assignment over months 
during MOS development, the remainder of 
the team could not regularly call on their 
expertise and support of overflow work. 

Furthermore, these volunteers spent 11 and 
16 months, respectively, displaced from 
home. 



Figure 4. MOS Operator Roles in Development. Most MOS staff 
played key roles in spacecraft, payload, CDS, or MOS procedure 
development. This simplified training, but contributed to a heavy 
workload and occasionally hindered simultaneous MOS and GDS 
development. 

flight) was assigned the responsibility of implementing the command sequences used by the spacecraft autonomy 
and fault management (A&FM) system for critical events, including initial spacecraft power-up and onboard fault 
responses. Working closely with Northrop Grumman systems engineers who designed the command sequences, this 
team member became expert with using the command and telemetry databases, onboard command sequence 
authoring and compilation, and the basic operation of onboard subsystems. 


D. MOS Team in LCROSS Payload Development 

The Payload Flight Software Lead for the science payload evolved naturally into the primary Payload Engineer 
for flight. In his development role, this team member designed and implemented all supplementary flight code for 
the operation of the payload, implemented all instrument command sequences for specific science activities, and 
also designed and implemented software used by the science team to analyze imagery and to assess payload 
throughput. He also supported payload testing. No other experience could have prepared this person so thoroughly 
for the position of Payload Engineer. 

One of the science team members also served as Payload Test Engineer during payload development, and took a 
key role in science operations from the SOC during flight. This test experience complemented her role as a mission 
scientist, and enabled her to provide additional oversight of payload performance during science activities. 

The only negative aspect of this arrangement was that when the timing of MOS test exercises conflicted with 
payload development and test activities, these two operators generally had to prioritize their payload responsibilities. 

E. MOS Team in GDS Development 

Several MOS team members either oversaw GDS development or developed software tools for their respective 
MOS subsystems. The GDS Lead, in charge of all LCROSS ground system development, became one of two Flight 
Directors. While larger teams might benefit from having dedicated software developers, the LCROSS MOS found 
distinct advantages in having its software end users also develop code. Most importantly, the problem of accurately 
communicating detailed requirements is obviated when the customer and developer is the same person. This also 
goes for software training - the software developer is the most familiar with a tool’s capabilities, limitations and 
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idiosyncrasies. This is especially important in flight, when deep software knowledge could enable an operator to 
work around a bug that might otherwise interfere with the support of an important mission event. Other MOS team 
members served as GDS team members, but in capacities distinct from their operational roles, with little training 
advantage. 

Despite these advantages, having dually-tasked operators had its share of disadvantages. For one, when 
developer and operator are distinct individuals, the developer can perform unit-level testing, then pass the software 
to the operator to perform independent requirements verification testing. When one person serves in both roles, 
another operator, often either less well-trained, or even from a different discipline, must perform independent 
verification testing. Furthermore, peak pre-launch workloads often volleyed between the GDS and MOS teams 
according to the interleaved GDS and MOS test cycles. For those working on both teams, the workload was 
extremely difficult and afforded them little rest time. Also, because MOS and GDS work often needed to happen 
concurrently (e.g. software development and MOS rehearsal preparations), one task or the other often suffered. 

These disadvantages continued in flight. For a number of reasons, LCROSS operations were on average more 
busy than anticipated. Inevitable GDS bug fixes and enhancements were developed during flight, but 
developer/operators were too consumed with flight duties to perform global GDS deployments. The team had to 
resort to less-formal point deployments for specific tools, complicating the GDS configuration management task. 

VII. Summary of Performance in Flight 

Arguably, the best measure of flight team preparation is its level of performance during its flight mission. This 
section provides a brief synopsis of LCROSS MOS performance during 112 days of flight, from a human 
standpoint. 

Transfer Phase was six days of 24-hour operations, covering launch through lunar swingby, and the most 
challenging nominal segment of flight, save the pre-impact sequence. The MOS operated in two overlapping shifts 
of 13 hours, synchronized with major events. The MOS successfully performed all nominal events, and 
simultaneously responded to several in-flight spacecraft anomalies that forced some significant operational changes 
from the baseline to maintain spacecraft health. On Day 3, a combination of a lack of situational awareness 
(stemming from one of the anomalies) and operator error caused the spacecraft to transition to a “safe mode”, with 
no lasting mission effect or major influence on other events. By the end of Transfer Phase, the team was noticeably 
fatigued, particularly Shift B which had worked graveyard hours. 

Cruise Phase, the bulk of time in flight, was more difficult than anticipated. With anomalies from Transfer Phase 
still present, the early part of Cruise was spent developing sustainable workarounds. The preparation and execution 
of nominal events also occupied far more time than expected. Due to late changes in the LCROSS launch date, 
Deep Space Network contacts were at highly variable times of day, and at variable intervals. Based on results from 
earlier science events, the operational profiles of later payload calibrations were modified, requiring extra team 
effort to accommodate. Furthermore, efforts to remove ice from the Centaur upper stage were less effective than 
expected, prompting the team to design and execute two additional maneuvers for that purpose. On the second Earth 
revolution, a substantial spacecraft anomaly caused the MOS to divert from nominal operations for two weeks to 
save remaining propellant and to guard against similar future problems. On emergence from this particularly taxing 
recovery period, the team progressed steadily towards a more sustainable operational cadence. The final weeks of 
the mission interleaved final rehearsals for Centaur separation and impact with nominal operations, and increased 
the frequency of trajectory maneuvering in preparation for impact. Despite these challenges, the MOS performed 
with very few operational errors, none of which had measurable negative influence on mission outcome. 

Impact Phase was executed nearly flawlessly. The MOS operated for roughly TBD hours continuously, and 
performed two full operational cycles including the execution of Centaur separation and lunar impact. There were 
some voice loop communications problems in the final minutes of flight, attributed to shortcomings in training for 
real-time instrument commanding. These had some effect on data collected at the time of the Centaur impact, but 
with a negligible influence on overall science goals. LCROSS met all of its mission objectives. 

VIII. Conclusions 

What were the keys to the success of the LCROSS flight team? Perhaps most importantly, LCROSS benefitted 
tremendously from a pervasive spirit of cooperation and trust that crossed organizational boundaries. This improved 
communications at all levels, spawned additional collaborations, and caused people to devote extra time throughout 
the project to ensure mission success. From the MOS development perspective, securing full-time access to the 
LCROSS simulator was critically important. It allowed the team to gain early, risk-free experience in the operation 
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of LCROSS, and provided the team with a realistic test platform in creating the hundreds of command products to 
support various mission events. The simulator was also the foundation for its series of MOS tests and rehearsals. 

Not surprisingly, frequent, repeated testing of the team under realistic conditions was invaluable. These tests 
exposed weaknesses throughout MOS development, enabling the team to perfect its equipment, processes, 
procedures, and staffing schedules before launch. By launch, the team had “flown” each major event multiple times, 
and this significantly improved team confidence in flight. As an unintended consequence, MOS simulator-based 
tests were partially responsible for exposing two significant spacecraft bugs prior to launch (subsequently corrected). 
MOS testing (with MOS-developed flight command sequences) provided a level of realism that could not easily be 
achieved in standard spacecraft system-level verification tests. 

Planning and execution of the flight was not perfect. The MOS was worked very hard for the greater part of the 
mission. A very challenging development phase, with accumulation of vast overtime hours, continued right up until 
launch. The flight was busier and the schedule more irregular than anticipated, contributing further to team fatigue. 
For the most part, the team was getting sufficient sleep to avoid human error, but time off to tend to personal matters 
was uncommon. Mainly due to lack of time, the team did not train a full set of backups for critical operational 
positions. Fortunately, the team did not falter on attendance or performance due to illness or accident. 

Distributed operations were largely successful. The MOS made substantial improvements to communications 
prior to launch, and operators from all disciplines made strong contributions to the team, despite being remotely 
situated. Flowever, communications and situational awareness could have improved even more. Some basic 
problems (e.g. poor room acoustics) hampered communications between ARC and NG ROC’s. Furthermore, the NG 
ROCs were tasked with partially conflicting roles, exacerbating the communications problems: one to support 
closely-coordinated procedure execution as extensions of the MOCR (demanding quiet, focus, and attentive voice 
loop participation), and the other to provide assessments of spacecraft performance (benefitting from offline group 
discussion). Recognizing these conflicts, we revised ROC protocols mid-flight, with noticeable improvement. 

As discussed throughout the paper, staff members that were dually-tasked as MOS operators and GDS 
developers were unable to fully satisfy the demands of both roles in flight. Fligh-priority MOS tasks consumed most 
operators’ time. Without a sufficient break in the busy flight schedule to perform testing, performing a formal 
global GDS deployment was seen as too risky. Instead, critical fixes were introduced on isolated workstations. 

Despite its success, it is debatable whether the LCROSS approach is an appropriate model for sustainable future, 
low-budget, high-risk operations. LCROSS had the advantage of being an extremely exciting mission with low 
initial expectations. The MOS successfully transformed into a highly-capable operations team, but this could not 
have happened without the extraordinary hours the team contributed from the start of development through impact. 
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